Skip to content

fix(review): verdict trust: only the engine blocks, policy from the base branch, allowForceTier - #1722

Open
RaghwendraSingh wants to merge 3 commits into
fix/review-noisefrom
fix/review-verdict-trust
Open

RaghwendraSingh wants to merge 3 commits into
fix/review-noisefrom
fix/review-verdict-trust

Conversation

@RaghwendraSingh

@RaghwendraSingh RaghwendraSingh commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Changes since review: Review feedback commit: blockingRules spelling, finding origin taken from evidence only, the policy hash computed after workflow flags, and a clear error when the base policy can't be read.

Issue for this PR

Closes #1792
Stacked on #1721, which is stacked on #1720 and #1719; merge in that order.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

The README and the docs promise "only the deterministic engine blocks". In the code, any three non-AI warnings, or one regex-catalog critical (join-no-on, create-or-replace, pii_into_mart, …), produced REQUEST_CHANGES. Separately, .altimate/review.yml was read from the PR's own checkout. So a PR could set mode: comment, drop a blockOn category, exclude its own paths, or narrow reviewers, and pass under its own relaxed rules.

  • Who can block: findings now carry origin: engine | catalog | ai, derived from evidence.tool. Unknown tools count as catalog, so a new detector can't block until someone classifies it. Only engine findings count toward REQUEST_CHANGES. A pattern rule a team trusts can be allowed to block via rubric.blockingRules: [<rule id>]. origin is stamped in buildEnvelope and covered by the signature.
  • Policy source: .altimate/review.yml is read at the merge-base commit (git show <base>:<path>). The PR's version is parsed only to list what it changes; those settings show in the comment and in policy.changedInPr and apply once merged. The envelope signs policy.source and policy.hash. Callers without git (pre-supplied files and content) still read the working tree.
  • --force-tier: allowForceTier: false in the base policy rejects the flag. Forced runs were already recorded as tier_forced in review_run.

Behavior changes:

  • Pattern-rule criticals and warning counts no longer block by default.
  • A PR's edits to review.yml no longer apply to that PR.
  • The tests that asserted the old behavior are updated: the two pii_into_mart regex-twin tests now expect COMMENT, and the verdict unit tests use engine evidence.

Not covered here: CLI flags in the workflow file (--mode, --severity) still override the policy. On pull_request events GitHub runs the workflow from the PR, so a PR could edit those flags too. Branch protection on .github/ (CODEOWNERS) is the usual control there.

How did you verify your code works?

  • New test/altimate/review-verdict-trust.test.ts (11 tests):
    • origin classification, including unknown tools
    • 1 catalog critical + 3 catalog warnings → COMMENT
    • blockingRules opt-in, and engine criticals and warning patterns still blocking
    • origin covered by the signature (tamper fails verification)
    • policy hash and change listing
    • on real git repos with GITHUB_BASE_REF: a PR relaxing mode/exclude/ai still judged in gate mode with the change shown; a PR adding review.yml not applied; an unchanged policy applied with no note; allowForceTier: false rejecting --force-tier
  • 1,883 tests across the review files and release gates pass. bun typecheck is clean.

Cost side effects

None. One extra local git show per run.

Screenshots / recordings

N/A (CLI).

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code


Summary by cubic

Makes the review verdict deterministic and tamper-proof: only the deterministic engine can block, and a PR can no longer relax the policy it is reviewed under.

Old behavior: any three non-AI warnings, or one regex-catalog critical, produced REQUEST_CHANGES, and .altimate/review.yml was read from the PR's own checkout. Now pattern-rule criticals and warning counts only comment unless rubric.blockingRules opts a catalog rule in, and the policy comes from the merge-base commit so a PR's edits are listed but only take effect after merge. allowForceTier: false in the base policy rejects --force-tier.

Behavior changes

  • Findings carry an origin (engine | catalog | ai) derived from evidence.tool; a supplied origin is ignored, and unknown tools count as catalog, so a new detector can't block until classified. Origin is stamped in the envelope and covered by the signature.
  • Only engine findings count toward REQUEST_CHANGES; the AI lane never blocks. blockingRules matches rule ids with either - or _ spelling.
  • A PR's changes to review.yml show in the comment and policy.changedInPr, apply after merge, and the applied policy's source and hash (after flag overrides) are signed into the envelope. Non-git callers still read the working tree; an unparsable base file fails closed.
  • --mode and --severity flags can only tighten a base policy: they can't turn mode: gate into comment or raise its severity threshold.

Written for commit b354b9f. Summary will update on new commits.

View guided diff Turn on auto-fix

@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: dd8277f2-9a6c-45be-83e3-34e446abe794

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown

Thanks for your contribution!

This PR doesn't have a linked issue. All PRs must reference an existing issue.

Please:

  1. Open an issue describing the bug/feature (if one doesn't exist)
  2. Add Fixes #<number> or Closes #<number> to this PR description

See CONTRIBUTING.md for details.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 issues found across 12 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/opencode/test/altimate/review.test.ts">

<violation number="1" location="packages/opencode/test/altimate/review.test.ts:145">
P2: This fixture update leaves the unknown-confidence regression test ineffective: `u()` has no engine evidence, so its findings are classified as catalog and excluded before confidence is evaluated. Add deterministic-engine evidence to `u()` (or reuse the engine fixture) so the test proves three unknown engine warnings cannot block.</violation>
</file>

<file name="packages/opencode/src/altimate/review/format.ts">

<violation number="1" location="packages/opencode/src/altimate/review/format.ts:221">
P2: This note misidentifies the policy file when the repository uses the supported `.altimate/review.yaml` or `.altimate/review.json` filename. Use a generic `review policy` label or carry the selected filename into the envelope before rendering.</violation>
</file>

<file name="packages/opencode/test/altimate/review-verdict-trust.test.ts">

<violation number="1" location="packages/opencode/test/altimate/review-verdict-trust.test.ts:115">
P3: These integration tests leak one temporary Git repository per case. Use the repository test fixture or remove each root in a `finally` block so repeated test runs do not accumulate temp directories.</violation>
</file>

<file name="packages/opencode/src/altimate/review/finding.ts">

<violation number="1" location="packages/opencode/src/altimate/review/finding.ts:107">
P1: The namespace prefix is too broad for the trust boundary: an unrecognized `altimate_core.*` tool is promoted to `engine` and can block. Replace the prefix check with an explicit allowlist of the known deterministic evidence tools so new tools remain catalog until classified.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

if (f.origin) return f.origin
const tool = f.evidence?.tool ?? ""
if (tool === "ai-review") return "ai"
if (tool.startsWith("altimate_core.") || ENGINE_TOOLS.has(tool)) return "engine"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: The namespace prefix is too broad for the trust boundary: an unrecognized altimate_core.* tool is promoted to engine and can block. Replace the prefix check with an explicit allowlist of the known deterministic evidence tools so new tools remain catalog until classified.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/finding.ts, line 107:

<comment>The namespace prefix is too broad for the trust boundary: an unrecognized `altimate_core.*` tool is promoted to `engine` and can block. Replace the prefix check with an explicit allowlist of the known deterministic evidence tools so new tools remain catalog until classified.</comment>

<file context>
@@ -84,9 +84,36 @@ export const Finding = z.object({
+  if (f.origin) return f.origin
+  const tool = f.evidence?.tool ?? ""
+  if (tool === "ai-review") return "ai"
+  if (tool.startsWith("altimate_core.") || ENGINE_TOOLS.has(tool)) return "engine"
+  return "catalog"
+}
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. Switch to an explicit allowlist of engine tools.

Comment thread docs/docs/usage/dbt-pr-review.md Resolved
notes.push(`${env.scope.outOfScopeFiles} changed file(s) outside the dbt project were not reviewed.`)
if (env.policy?.changedInPr?.length) {
notes.push(
`This PR changes the review policy (\`.altimate/review.yml\`: ${env.policy.changedInPr.join(", ")}). ` +

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This note misidentifies the policy file when the repository uses the supported .altimate/review.yaml or .altimate/review.json filename. Use a generic review policy label or carry the selected filename into the envelope before rendering.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/format.ts, line 221:

<comment>This note misidentifies the policy file when the repository uses the supported `.altimate/review.yaml` or `.altimate/review.json` filename. Use a generic `review policy` label or carry the selected filename into the envelope before rendering.</comment>

<file context>
@@ -216,6 +216,12 @@ export function coverageNotes(env: VerdictEnvelope): string[] {
     notes.push(`${env.scope.outOfScopeFiles} changed file(s) outside the dbt project were not reviewed.`)
+  if (env.policy?.changedInPr?.length) {
+    notes.push(
+      `This PR changes the review policy (\`.altimate/review.yml\`: ${env.policy.changedInPr.join(", ")}). ` +
+        "The base branch's policy was applied; the change takes effect after merge.",
+    )
</file context>
Suggested change
`This PR changes the review policy (\`.altimate/review.yml\`: ${env.policy.changedInPr.join(", ")}). ` +
`This PR changes the review policy (${env.policy.changedInPr.join(", ")}). ` +

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. Say "review policy" generically.

Comment thread docs/docs/usage/dbt-pr-review.md Resolved
})

async function repo(baseYml: string | undefined, headYml: string | undefined) {
const root = await fs.realpath(await fs.mkdtemp(path.join(tmpdir(), "review-policy-")))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: These integration tests leak one temporary Git repository per case. Use the repository test fixture or remove each root in a finally block so repeated test runs do not accumulate temp directories.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/test/altimate/review-verdict-trust.test.ts, line 115:

<comment>These integration tests leak one temporary Git repository per case. Use the repository test fixture or remove each root in a `finally` block so repeated test runs do not accumulate temp directories.</comment>

<file context>
@@ -0,0 +1,174 @@
+  })
+
+  async function repo(baseYml: string | undefined, headYml: string | undefined) {
+    const root = await fs.realpath(await fs.mkdtemp(path.join(tmpdir(), "review-policy-")))
+    await g(root, "init", "-q", "-b", "main")
+    await g(root, "config", "user.email", "t@test")
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing in this PR: test-hygiene nit.


/** File content at a commit (`git show <ref>:<path>`), or undefined when the
* path does not exist there. `rel` is repository-relative, POSIX. */
export async function showAtRef(cwd: string, ref: string, rel: string): Promise<string | undefined> {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: A failed git show is silently treated as "no config at the base commit"

readFirst treats the undefined returned here as "candidate absent" and moves on, so loadReviewPolicy falls back to DEFAULT_REVIEW_CONFIG (mode: comment, allowForceTier: true) and stamps policy.source: "default" — indistinguishable from "the base branch has no config". Any git-level failure (bad object, missing shallow history, permissions, git erroring for a reason other than the path being absent) therefore silently demotes a gate policy to the permissive default instead of surfacing it, which is the opposite of the trust property this PR adds. git cat-file -e <ref>:<path> (or checking the exit code/stderr) would distinguish "path missing at that commit" from "git failed" and let the latter throw.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 956f4ef (see the cubic thread on showAtRef).

if (origin === "engine") return true
if (origin === "ai") return false
const rule = catalogRuleId(f)
return !!rule && rubric.blockingRules.includes(rule)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: blockingRules entries are matched by exact string and never validated

Rule ids are emitted by two catalogs with different conventions (rule-catalog uses hardcoded-credential, dbt-patterns uses pii_into_mart), and nothing checks that a configured id exists. A typo, a case difference, or the other convention therefore leaves the gate silently non-blocking — the exact failure mode a team opting a rule in wants to avoid — with no error and no note in the comment. Consider validating rubric.blockingRules against the known catalog ids when the rubric is resolved (warning when an entry matches nothing) and documenting where teams can find the ids.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. Warn on ids that match no catalog rule.

changedInPr: string[]
}

function canonical(value: unknown): string {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: canonical duplicates stableStringify in verdict.ts

verdict.ts:278 already implements this exact algorithm (keys sorted at every depth, array order preserved, undefined dropped) for the signature body. Export that helper (or move it to a shared module) and use it for policyHash too: policy.hash is signed alongside the signature body, so the two canonicalizations should stay identical by construction instead of being two copies that can drift.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing: nit. They serve different modules; consolidation can happen in a cleanup pass.

}): Promise<ReviewPolicy> {
if (!opts.readBase) {
const config = await loadReviewConfig(opts.cwd)
const present = await readFirst((rel) => fs.readFile(path.join(opts.cwd, rel), "utf8").catch(() => undefined))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: The no-git branch reads the same candidate file twice

loadReviewConfig above has already scanned CANDIDATE_FILES and read the first existing file; this second readFirst repeats the same read sequence only to learn whether a file was present, and the candidate loop is now written a third time here. Folding both into one helper that returns { text, rel } | undefined (and parsing that text once) removes the extra I/O and the chance of the two passes disagreeing if the file changes between them.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing: nit, one extra small file read on a non-default path.

const out: string[] = []
const keys = new Set([...Object.keys(base), ...Object.keys(head)]) as Set<keyof ReviewConfig>
for (const k of [...keys].sort()) {
if (canonical(base[k]) === canonical(head[k])) continue

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Comparing parsed partial configs reports settings spelled out at their default values as changes

Representation, not effective policy, drives this comparison. A base file with mode: gate parses to rubric: {}; a PR that adds rubric:\n thresholds:\n warehouseCostMinRows: 1000000 (the default) parses to a full rubric.thresholds object, so the PR comment claims it "changes the review policy (rubric.thresholds)" that "takes effect after merge" even though the resolved rubric is identical. Comparing resolveRubric(...) results (and normalized top-level values) instead of the raw partials would keep the note to genuine behavior changes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked. Compare resolveRubric() outputs.

throw new Error("--force-tier is disabled by `allowForceTier: false` in the base branch's .altimate/review.yml")
}
if (opts.manifestPath) config.manifestPath = opts.manifestPath
if (opts.mode) config.mode = opts.mode

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: policy.hash is computed before these overrides, so it can describe a different policy than the envelope

loadReviewPolicy hashes the file config, then this line and the severityThreshold override mutate config, and the envelope later records policy.hash next to mode: config.mode / severityThreshold. With --mode gate (which github/review/action.yml always passes) the signed hash says comment while the signed mode says gate, so an auditor cannot check that the hashed policy is the one that was applied. Hashing after the overrides, or documenting that the hash covers only the config file, would remove the ambiguity.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked (same as the cubic thread).

title: "t",
body: "b",
file: `m_${Math.random()}.sql`,
evidence: { tool: "altimate_core.check", result: {} },

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: This makes the unknown-confidence test pass for the wrong reason

Adding engine evidence to mk is right for the blocking assertions, but "unknown-confidence warnings do NOT accumulate into a block" (u() around line 168) builds findings with raw makeFinding and no evidence. Those are now catalog origin and ineligible regardless of confidence, so the f.confidence !== "unknown" clause in computeIdealVerdict is no longer exercised by that test. Giving u() engine evidence (altimate_core.equivalence) keeps it guarding the confidence rule.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: low priority, tracked (same as the cubic thread).

### Which policy applies, and what can block

- **The base branch's `.altimate/review.yml` applies.** The file is read from the
commit the PR branched from, so a PR cannot relax the rules it is reviewed

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: The docs state a stronger trust boundary than the shipped path enforces

The policy file is base-scoped, but github/review/action.yml always passes --mode/--severity from the workflow (args=(--mode "$IN_MODE" ...)), and on pull_request events that workflow comes from the PR. A PR can therefore still switch the gate to comment mode by editing .github/workflows/*.yml — the PR description notes this as out of scope, but this bullet reads as if no PR-side edit can relax the gate. Adding that caveat (and pointing at CODEOWNERS / branch protection on .github/) would keep the doc accurate.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 956f4ef. Workflow flags can no longer relax a base policy, and the docs state the remaining caveat (protect .github/).

@kilo-code-bot

kilo-code-bot Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Code Review Summary

Status: 5 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 0
SUGGESTION 5

Review scope: merge-base 4726770f9 → head b354b9fe (12 files). The previous round's two WARNINGs are fixed and verified: showAtRef now returns undefined only for git's "path absent" errors and rethrows everything else, and blockingRules ids compare through normalizeRuleId so either catalog spelling matches. The documented trust properties also check out: origin is derived only from evidence.tool (a supplied origin is ignored by computeIdealVerdict), buildEnvelope stamps it before signEnvelope so it is signature-covered, the AI lane still hardcodes tool: "ai-review" and can never be verdict-eligible, the policy is read from the merge-base commit via showAtRef with a fail-closed parse, and the re-ordered policy.hash is computed after the --mode/--severity clamp. --severity may be lowered (surfacing more) but not raised, which matches the docs.

Issue Details (click to expand)

SUGGESTION

File Line Issue
packages/opencode/test/altimate/review-verdict-trust.test.ts 189 The new "--mode and --severity ... only tighten" test asserts only mode; the severity clamp (passing critical against a base warning) is never verified
packages/opencode/src/altimate/review/config.ts 133 (previously reported, still present) canonical duplicates stableStringify in verdict.ts
packages/opencode/src/altimate/review/config.ts 194 (previously reported, still present) the no-git branch reads the same candidate config file twice
packages/opencode/src/altimate/review/config.ts 158 (previously reported, still present) policyChanges compares parsed partials, so settings spelled at default values are reported as policy changes
packages/opencode/test/altimate/review.test.ts 145 (previously reported, still present) the unknown-confidence test's u() helper still has no engine evidence, so it passes because those findings are ineligible catalog, not because of confidence

The four carried suggestions are on lines outside the PR's hunk-level additions and already have active inline comments, so they are summary-only here.

Files Reviewed (12 files)
  • docs/docs/usage/dbt-pr-review.md - 0 issues
  • packages/opencode/src/altimate/review/config.ts - 3 issues (carried)
  • packages/opencode/src/altimate/review/finding.ts - 0 issues
  • packages/opencode/src/altimate/review/format.ts - 0 issues
  • packages/opencode/src/altimate/review/git.ts - 0 issues
  • packages/opencode/src/altimate/review/orchestrate.ts - 0 issues
  • packages/opencode/src/altimate/review/rubric.ts - 0 issues
  • packages/opencode/src/altimate/review/run.ts - 0 issues
  • packages/opencode/src/altimate/review/verdict.ts - 0 issues
  • packages/opencode/test/altimate/review-verdict-trust.test.ts - 1 issue
  • packages/opencode/test/altimate/review.test.ts - 1 issue (carried)
  • packages/opencode/test/skill/release-v0.8.1-adversarial.test.ts - 0 issues

Notes / assumptions: this is a read-only review, so tests were not executed; the test-quality finding is derived from reading the assertions. Known residual gaps are documented in the PR and the docs (a caller that supplies both changedFiles and getContent gets the working-tree policy, and other workflow steps remain PR-editable).

Fix these issues in Kilo Cloud

Previous Review Summary (commit 8ce9cec)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 8ce9cec)

Status: 8 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 6

The design is sound and the core claims check out: origin is derived from evidence.tool (altimate_core.* plus impact_analysis / dbt.manifest / data.diff / schema.detect_pii), every engine lane that emits findings sets evidence, the AI lane still hardcodes tool: "ai-review" and never sets origin (so it cannot become verdict-eligible), buildEnvelope stamps origin before signEnvelope so it is covered by the signature, and both shipped callers (cli/cmd/review.ts, tools/dbt-pr-review.ts) resolve a base, so they now read the policy from the merge-base commit rather than the PR checkout.

Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/git.ts 266 showAtRef swallows every git error as "path absent", so a git failure silently falls back to the default policy (mode comment, allowForceTier: true) instead of surfacing the failure
packages/opencode/src/altimate/review/verdict.ts 52 blockingRules ids are matched by exact string with no validation; ids come from two catalogs with different conventions, so a typo silently leaves the gate non-blocking

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/config.ts 133 canonical duplicates stableStringify from verdict.ts; reuse one implementation for the hash and the signature
packages/opencode/src/altimate/review/config.ts 194 No-git branch reads the same candidate config file twice; fold into a single read that also reports presence
packages/opencode/src/altimate/review/config.ts 158 policyChanges compares parsed partials, so settings spelled out at default values (e.g. rubric.thresholds) are reported as policy changes
packages/opencode/src/altimate/review/run.ts 261 policy.hash is computed before --mode / --severity overrides, so the signed hash can disagree with the signed mode
packages/opencode/test/altimate/review.test.ts 145 Adding engine evidence to mk makes the unknown-confidence test pass because those findings are now ineligible catalog, not because of confidence
docs/docs/usage/dbt-pr-review.md 289 Docs omit that workflow-level --mode / --severity flags (from the PR-editable workflow file) still override the base policy

Notes / assumptions: this review found no path where the new code makes a non-engine finding block, and the AI lane remains non-blocking. The documented residual gap (pre-supplied changedFiles + getContent, and CLI flags in the workflow) is consistent with the PR description; only the missing user-doc caveat is flagged. Tests could not be executed in this read-only review, so the test-quality finding is derived from reading the assertions.

Files Reviewed (12 files)
  • docs/docs/usage/dbt-pr-review.md - 1 issue
  • packages/opencode/src/altimate/review/config.ts - 3 issues
  • packages/opencode/src/altimate/review/finding.ts - 0 issues
  • packages/opencode/src/altimate/review/format.ts - 0 issues
  • packages/opencode/src/altimate/review/git.ts - 1 issue
  • packages/opencode/src/altimate/review/orchestrate.ts - 0 issues
  • packages/opencode/src/altimate/review/rubric.ts - 0 issues
  • packages/opencode/src/altimate/review/run.ts - 1 issue
  • packages/opencode/src/altimate/review/verdict.ts - 1 issue
  • packages/opencode/test/altimate/review-verdict-trust.test.ts - 0 issues
  • packages/opencode/test/altimate/review.test.ts - 1 issue
  • packages/opencode/test/skill/release-v0.8.1-adversarial.test.ts - 0 issues

Fix these issues in Kilo Cloud


Reviewed by deepseek-v4.1-flash-1017 · Input: 336.4K · Output: 81.6K · Cached: 7.3M

Review guidance: REVIEW.md from base branch fix/review-noise

@RaghwendraSingh
RaghwendraSingh force-pushed the fix/review-verdict-trust branch from 8ce9cec to 4004ec6 Compare November 13, 2026 06:36
@kilo-code-bot

kilo-code-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

Kilo Code Review could not run — your account is out of credits.

Add credits or switch to a free model to enable reviews on this change.

@hkaltai hkaltai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: approve (no CRITICAL/MAJOR). Panel: Claude, Codex, lead. The design is sound: unclassified tools default to catalog, so new detectors cannot block until classified; the AI lane stays non-blocking; the policy is read from the merge-base and signed in the envelope. I could not run the Bun suite in the review worktree (dependencies not installed); findings were verified by reading the code and isolated probes.

Counts: CRITICAL 0, MAJOR 0, MINOR 6, NIT 6.

Also worth a changelog/release note: pattern-rule criticals (cross_join, pii_into_mart, etc.) and non-engine warning counts stop blocking by default, and local edits to review.yml no longer apply to the PR being reviewed. Missing tests: hyphen/underscore blockingRules ids, hash after CLI overrides, the no-git path (changedFiles + getContent), a cwd in a repo subdirectory, base-read failure and an invalid base file.

Nits

  • finding.ts ENGINE_TOOLS (~L96): altimate_core.lexical is a pre-parse lexical check and schema.detect_pii is text-based, yet both count as "engine"; the comment "(AST, manifest, lineage, warehouse)" overstates it. Fix: adjust the comment or classify them explicitly.
  • config.ts canonical() (~L136): duplicates stableStringify in verdict.ts. Fix: export and reuse it.
  • verdict.ts computeIdealVerdict doc: "the docs promise that only the deterministic engine blocks" references docs instead of stating the invariant. Fix: state the invariant directly.
  • finding.ts catalogRuleId: also returns ids for engine findings. Fix: rename to ruleIdOf.
  • rubric.ts blockingRules: never affects engine findings. Fix: consider blockingCatalogRules.
  • format.ts (~L24-36): a non-blocking catalog critical renders as "Reviewed with comments, 1 critical" with no hint why it did not block. Fix: add a footnote when origin filtering lowered the verdict.

if (origin === "engine") return true
if (origin === "ai") return false
const rule = catalogRuleId(f)
return !!rule && rubric.blockingRules.includes(rule)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] blockingRules matches ids by exact string; catalog sources spell ids differently

  return !!rule && rubric.blockingRules.includes(rule)

rule-catalog ids are hyphenated (hardcoded-credential) while dbt-patterns ids use underscores (pii_into_mart, cross_join); orchestrate.ts already normalizes _ to - when comparing ids. Failure: blockingRules: ["pii-into-mart"] yields COMMENT, ["pii_into_mart"] yields REQUEST_CHANGES, so a team opting a rule in with the other spelling silently gets no blocking.

Fix: compare after normalizing _ to - on both sides, and list valid ids in the docs or warn on unknown ids at config load. (Claude, lead; premise-checked)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks. Fixed in fix(review): address review feedback: blockingRules spelling, origin from evidence only, hash after flags, base-policy error. Ids compare through normalizeRuleId (lowercase, _ as -) on both sides, so pii-into-mart and pii_into_mart both opt the rule in. Tested. Warning on unknown ids at config load is still a follow-up.

const cwdRel = await repoRelative(gitRoot, opts.cwd, true)
const policy = await loadReviewPolicy({
cwd: opts.cwd,
readBase:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] Callers supplying changedFiles + getContent get no base-policy protection

    readBase:
      resolved && cwdRel !== undefined

When both are supplied, needGit is false, resolved is undefined and the working-tree policy applies, even if base was passed explicitly. The PR description states this limit, so I rate it MINOR (Codex rated MAJOR; downgraded since it is documented and the git-less path has no trusted base to read). Scenario: an integration with pre-computed diffs reviews a PR whose review.yml sets mode: comment and it is honored.

Fix: let such integrations supply a trusted readBase/policy text, and emit a visible note when source is working_tree in a PR context. (Codex)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I'm not changing this in this PR. That path exists for integrations without git history, so there is no trusted base to read, and the CLI and the action always resolve git. It is documented in the PR description. A caller-supplied readBase and a visible note for source: working_tree in a PR context are the right follow-up, and they are tracked.

throw new Error("--force-tier is disabled by `allowForceTier: false` in the base branch's .altimate/review.yml")
}
if (opts.manifestPath) config.manifestPath = opts.manifestPath
if (opts.mode) config.mode = opts.mode

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] policy.hash is computed before CLI overrides

  if (opts.mode) config.mode = opts.mode

The hash is taken inside loadReviewPolicy, then --mode/--severity/--manifest mutate the same config. envelope.policy.hash therefore describes the base file, not the policy actually applied (mode is signed separately, but severityThreshold and manifestPath overrides are recorded nowhere). Repro: base mode: gate hash differs from the hash of the post---mode comment config.

Fix: recompute the hash after overrides, or record an overrides list beside it, or document the hash as the base-file policy. (Claude, Codex)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, thanks. Fixed in the same commit: the hash is recomputed after --mode/--severity/--manifest, so it describes the applied policy. There's a test that a tightened run has a different hash.

* path does not exist there. `rel` is repository-relative, POSIX. */
export async function showAtRef(cwd: string, ref: string, rel: string): Promise<string | undefined> {
try {
return await git(["show", `${ref}:${rel}`], cwd)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] showAtRef treats every git error as "file absent"

    return await git(["show", `${ref}:${rel}`], cwd)

A failure other than a missing path (missing object in a partial clone, permissions) silently yields default policy (source: "default"), dropping a strict base policy with no message.

Fix: return undefined only for git's "does not exist in" / "exists on disk, but not in" errors; rethrow otherwise. (Claude)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Fixed in fix(review): address review comments: workflow flags cannot relax base policy, showAtRef fails closed: only git's "does not exist in" / "exists on disk, but not in" map to absent, and anything else is rethrown. Tests cover a missing path, a present path and a bad ref.

try {
config = baseText === undefined ? structuredClone(DEFAULT_REVIEW_CONFIG) : parseReviewConfig(baseText)
} catch (e: any) {
throw new Error(`Failed to load the base branch's .altimate/review config: ${e?.message ?? e}`)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] An unparsable base review.yml blocks the PR that would fix it

    throw new Error(`Failed to load the base branch's .altimate/review config: ${e?.message ?? e}`)

If the base file is invalid, every review throws, including the fix PR, so recovery needs an admin bypass. Previously the fix PR could repair itself. Failing closed is defensible; consider a clearer message naming the recovery path, or falling back to the PR's parsable version with a loud note.

(Claude)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point, thanks. Failing closed stays, so a PR can never relax its own policy by breaking the base file. In the same commit the error now names the recovery path: fix the file on the base branch, or merge the repair PR with an admin bypass. Tested.

/** The layer a finding came from. Unknown or missing evidence counts as
* `catalog`, so a new detector can never block until it is classified. */
export function findingOrigin(f: Pick<Finding, "evidence" | "origin">): FindingOrigin {
if (f.origin) return f.origin

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] An explicit origin overrides evidence.tool

  if (f.origin) return f.origin

The PR says origin is derived from evidence.tool, but a pre-set value wins. Codex showed three ai-review warnings carrying origin: "engine" yielding REQUEST_CHANGES. No in-repo producer sets origin (ai-review builds findings field by field), so this is only reachable via externally supplied findings.

Fix: always derive from evidence, or reject mismatched supplied origins. (Codex; premise-checked)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, thanks. Fixed in the same commit: findingOrigin always derives from evidence.tool, and buildEnvelope re-derives it for every finding. Three AI warnings with origin: "engine" now give COMMENT (tested).

RaghwendraSingh and others added 3 commits November 20, 2026 12:11
…ase branch, `allowForceTier`

- Findings carry `origin: engine | catalog | ai` (from `evidence.tool`;
  unknown tools count as catalog). Only engine findings count toward
  REQUEST_CHANGES, matching the documented promise; a pattern rule can be
  opted in by id via `rubric.blockingRules`. The AI lane never blocks.
- `.altimate/review.yml` is read from the merge-base commit, so a PR cannot
  relax the rules it is judged by. Settings the PR changes are listed in the
  comment and envelope (`policy.changedInPr`) and take effect after merge.
  The applied policy's source and hash are signed into the envelope.
- `allowForceTier: false` in the base policy rejects `--force-tier`
  (forced runs were already recorded as `tier_forced` in `review_run`).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e policy, `showAtRef` fails closed

- `--mode` / `--severity` (always passed by the action from the PR-editable
  workflow) can only tighten a base-branch policy: `mode: gate` is kept and a
  higher severity threshold is ignored, each with a stderr notice. Docs state
  this and recommend protecting `.github/` with CODEOWNERS.
- `showAtRef` returns undefined only when the path is absent at the ref and
  rethrows any other git failure, so the base-policy loader no longer falls
  back to the permissive default on a bad ref or git error.
- Tests for both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n from evidence only, hash after flags, base-policy error

- `verdictEligible`: `blockingRules` ids compare after `normalizeRuleId`
  (lowercase, `_` as `-`), so `pii-into-mart` and `pii_into_mart` both opt the
  rule in.
- `findingOrigin` always derives from `evidence.tool` and `buildEnvelope`
  always re-derives it: a supplied `origin` cannot make an AI or catalog
  finding block.
- `reviewPullRequest`: the signed `policy.hash` is recomputed after the
  `--mode`/`--severity`/`--manifest` flags, so it describes the applied policy.
- An unparsable base `review.yml` still fails closed; the error now names the
  recovery path.
- Tests: both id spellings, a supplied `origin: "engine"` on AI findings, hash
  after flags, unparsable base policy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RaghwendraSingh

Copy link
Copy Markdown
Contributor Author

@hkaltai, re your review:

Thanks for the approval. One commit on top:

  • Fixed: blockingRules id spelling, origin always from evidence, hash after CLI overrides, and a clearer error for an unparsable base policy. showAtRef already failed closed.
  • Not changed: the git-less changedFiles + getContent path. It is documented, and a caller-supplied policy reader is a tracked follow-up. The mode/severity design stays tighten-only, as decided.
  • Tests added: hyphen/underscore blockingRules, hash after overrides, supplied origin, invalid base file. The no-git path and a cwd in a subdirectory are still open test follow-ups.
  • Changelog note, as you suggested: pattern-rule criticals no longer block by default (opt in with blockingRules), and local review.yml edits don't apply to the PR under review. This will go in the release notes.
  • Nits deferred: renaming catalogRuleId/blockingRules, sharing stableStringify, the ENGINE_TOOLS comment, and a footnote when origin filtering lowered the verdict.

test("--mode and --severity from the workflow cannot relax a base policy, only tighten it", async () => {
const root = await repo("mode: gate\nseverityThreshold: warning\n", undefined)
const relaxed = await reviewPullRequest({ cwd: root, head: "HEAD", noAi: true, mode: "comment", severityThreshold: "critical" })
expect(relaxed.mode).toBe("gate")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: The --severity half of this test is never asserted

The test is named for both --mode and --severity, and line 188 passes severityThreshold: critical against a base policy of warning — the exact vector where a PR-editable workflow could hide warnings. But the only assertion is on mode, so the clamp at run.ts (SEVERITY_ORDER[opts.severityThreshold] > SEVERITY_ORDER[config.severityThreshold]) is unverified: it could be changed to always apply the flag and this test would still pass. Either build the fixture so the head change yields a warning finding and assert it survives --severity critical, or record the applied threshold in the signed envelope (policy) and assert it there — the latter would also make the effective threshold auditable, which today only the opaque hash covers.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants